Incremental All Pairs Similarity Search for Varying Similarity Thresholds with Reduced I/O Overhead

All Pairs Similarity Search (APSS) is the problem of finding all pairs of records with similarity scores above a specified threshold. Incremental All Pairs Similarity Search (IAPSS) is the problem of performing APSS multiple times over the same dataset by varying the similarity threshold. This problem is ubiquitous in many real-world systems like search engines, online social networks, and digital libraries. A significant part of the computation is redundant across multiple invocations of APSS. Our solution to the IAPSS problem avoids these redundant computations by storing the history of previous APSS invocations and splitting the inverted index that maps each dimension into a list of records that have non-zero projections along that dimension. The size of the computation history increases quadratically with the number of records in the dataset. We introduce the concept of a similarity floor to store partial computation history, resulting in reduced I/O overhead. We empirically evaluate the effectiveness of our techniques using four real-world large-scale datasets. Our IAPSS solution achieves speed-ups in the order of 2X to over 105 X over the state-of-the-art APSS algorithm, while reducing the size of the computation history by at least an order of magnitude.